Identifying alarms for a root cause of a problem in a data processing system

ABSTRACT

Methods for monitoring a networked computing environment and for consolidating multiple alarms under a single root cause are described. In some embodiments, in response to detecting an alert corresponding with a performance issue in a networked computing environment, a root cause identification tool may aggregate a plurality of alarms from a plurality of performance management tools monitoring the networked computing environment. The root cause identification tool may then generate a failure graph associated with the performance issue based on the plurality of alarms, determine a first set of leaf nodes of the failure graph, determine a first chain of failures based on the first set of leaf nodes, suppress (or hide) alarms that are not associated with the first chain of failures, and output a consolidated alarm associated with the first chain of failures.

BACKGROUND

This disclosure relates to systems and methods for monitoring anetworked computing environment and for identifying and reporting rootcauses of performance and availability issues that occur throughoutmultiple layers of the networked computing environment.

Information technology management software may refer to software toolsfor monitoring the performance and availability of resources andservices across a networked computing environment, such as a data centerenvironment or a cloud computing environment. The multiple layers of anetworked computing environment may be monitored and managed usingvarious software tools, such as application performance managementtools, network performance management tools, and system performancemanagement tools. Application performance management tools may monitorand manage the performance and availability of software applications.For example, an application performance management tool may monitor thevolume of transactions processed by a particular application over time(e.g., the number of transactions per second), the response times of theparticular application over various load conditions (e.g., averageresponse times under a peak load), and the computing and storageresources consumed by the particular application over time (e.g., thememory footprint). Network performance management tools may monitor theperformance of network resources and components, such as network routersand switches. For example, a network performance management tool mayidentify devices located on a network, monitor response times andavailability of devices connected to the network, and track bandwidthutilization and latency across the various components of the network.System performance management tools may monitor the performance ofcomputing devices and/or storage devices. For example, a systemperformance management tool may monitor CPU, memory, or disk usage overtime.

BRIEF SUMMARY

According to one aspect of the present disclosure, methods formonitoring a networked computing environment and for consolidatingmultiple alarms under a single root cause are disclosed.

In some embodiments, in response to detecting an alert correspondingwith a performance issue in a networked computing environment, a rootcause identification tool may aggregate a plurality of alarms from aplurality of performance management tools monitoring the networkedcomputing environment. The root cause identification tool may thengenerate a failure graph associated with the performance issue based onthe plurality of alarms, determine a first set of leaf nodes of thefailure graph, determine a first chain of failures based on the firstset of leaf nodes, suppress (or hide) alarms that are not associatedwith the first chain of failures, and output a consolidated alarmassociated with the first chain of failures.

In some embodiments, the networked computing environment may comprise anIT infrastructure that provides data processing and/or data storageservices. For example, the networked computing environment may comprisea portion of a data center that provides software services to end usersaccessing the networked computing environment. The networked computingenvironment may include resources (e.g., servers and storage units) andthe services provided by the networked computing environment (e.g.,web-based work productivity applications or business tools for managinga corporation's financial information). The networked computingenvironment may include servers, data storage units, power distributionunits, power delivery systems, cooling systems, virtualization layers,and the networking fabric connecting the servers and data storage unitsto each other and to external sources (e.g., an edge server connected toan ISP). The different components of the networked computingenvironment, including the power delivery systems, cooling systems,storage units, servers, applications, network connections, and servicesmay be monitored from different points of view using a plurality ofinformation technology management software tools that report performanceand availability metrics associated with the different components beingmonitored over time. In some embodiments, a root cause identificationtool may aggregate data from a plurality of information technologymanagement software tools monitoring different layers of a networkedcomputing environment, identify causal relationships associated with aperformance issue based on the aggregated data, determine a root causeof the performance issue based on the causal relationships, and reportthe root cause of the performance issue and possible remedies to theperformance issue to an end user of the root cause identification toolbased on a role of the end user.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The claimed subject matter is not limited to implementationsthat solve any or all disadvantages noted in the Background.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example andare not limited by the accompanying figures with like referencesindicating like elements.

FIG. 1 depicts one embodiment of a networked computing environment.

FIG. 2 depicts one embodiment of a portion of a data center.

FIG. 3 depicts one embodiment of a monitoring system including a rootcause identification manager for identifying a root cause of aperformance or availability issue affecting a networked computingenvironment.

FIG. 4 depicts one embodiment of an infrastructure mapping for mappingservices or applications to components of a networked computingenvironment that support the services or applications at a particulartime.

FIG. 5A depicts one embodiment of a failure graph template.

FIG. 5B depicts one embodiment of a failure graph generated using thefailure graph template of FIG. 5A.

FIG. 6A is a flowchart describing one embodiment of a process foridentifying a root cause of a failure or performance-related issuewithin a networked computing environment.

FIG. 6B is a flowchart describing an alternative embodiment of a processfor identifying a root cause of a failure or performance-related issuewithin a networked computing environment.

FIG. 6C is a flowchart describing one embodiment of a process forgenerating a failure graph.

FIG. 7A is a flowchart describing one embodiment of a process forconsolidating multiple alarms generated from a plurality of monitoringapplications monitoring a networked computing environment under a singleroot cause.

FIG. 7B is a flowchart describing one embodiment of a process foridentifying a single root cause for multiple failures occurring in anetworked computing environment.

FIG. 8A is a flowchart describing one embodiment of a process forvisualizing and accessing data associated with causal relationshipsbetween failures occurring within a networked computing environment.

FIG. 8B is a flowchart describing one embodiment of a process forautomatically generating a report in response to detecting performanceand/or availability issues that occur throughout multiple layers of anetworked computing environment based on a role of a target recipient ofthe report.

FIG. 9 is a flowchart describing one embodiment of a process forautomatically generating help desk tickets in response to detectingperformance and/or availability issues that occur throughout multiplelayers of a networked computing environment.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be illustrated and described herein in any of a number ofpatentable classes or context including any new and useful process,machine, manufacture, or composition of matter, or any new and usefulimprovement thereof. Accordingly, aspects of the present disclosure maybe implemented entirely hardware, entirely software (including firmware,resident software, micro-code, etc.) or combining software and hardwareimplementation that may all generally be referred to herein as a“circuit,” “module,” “component,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productembodied in one or more computer readable media having computer readableprogram code embodied thereon.

Any combination of one or more computer readable media may be utilized.The computer readable media may be a computer readable signal medium ora computer readable storage medium. A computer readable storage mediummay be, for example, but not limited to, an electronic, magnetic,optical, electromagnetic, or semiconductor system, apparatus, or device,or any suitable combination of the foregoing. More specific examples (anon-exhaustive list) of the computer readable storage medium wouldinclude the following: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an appropriateoptical fiber with a repeater, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable signal medium may be transmitted usingany appropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, CII, VB.NETor the like, conventional procedural programming languages, such as the“C” programming language, Visual Basic, Fortran 2003, Perl, Python,COBOL 2002, PHP, ABAP, dynamic programming languages such as Python,Ruby and Groovy, or other programming languages. The program code mayexecute entirely on the user's computer, partly on the user's computer,as a stand-alone software package, partly on the user's computer andpartly on a remote computer or entirely on the remote computer orserver. In the latter scenario, the remote computer may be connected tothe user's computer through any type of network, including a local areanetwork (LAN) or a wide area network (WAN), or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider) or in a cloud computing environment oroffered as a service such as a Software as a Service (SaaS).

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatuses(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that when executed can direct a computer, otherprogrammable data processing apparatus, or other devices to function ina particular manner, such that the instructions when stored in thecomputer readable medium produce an article of manufacture includinginstructions which when executed, cause a computer to implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable instruction execution apparatus, or otherdevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatuses or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Technology is described for monitoring a networked computing environmentand for identifying root causes of performance and/or availabilityissues that occur throughout multiple layers of the networked computingenvironment. The networked computing environment may include resourceswithin a data center (e.g., servers and storage units) and the servicesprovided by the data center (e.g., web-based work productivityapplications or business tools for managing a corporation's financialinformation). The networked computing environment may include servers,data storage units, power distribution units, power delivery systems,cooling systems, virtualization layers, and the networking fabricconnecting the servers and data storage units to each other and toexternal sources (e.g., an edge server connected to an ISP). In oneexample, the networked computing environment may comprise hundreds ofdata storage units in communication with thousands of servers runninghundreds of thousands of applications providing hundreds of softwareservices. The storage units, servers, applications, and services may bemonitored from different points of view using a plurality of informationtechnology management software tools that report performance andavailability metrics associated with the resources and applicationsbeing monitored over time. The networked computing environment may alsocomprise electrical systems for powering the servers and data storageunits, and cooling systems for regulating the ambient temperature nearthe servers and/or data storage units. In some embodiments, a root causeidentification tool may aggregate data from a plurality of informationtechnology management software tools, identify causal relationshipsassociated a performance issue based on the aggregated data, determine aroot cause of the performance issue based on the causal relationships,and report the root cause and possible remedies to the performance issueto an end user of the root cause identification tool based on a role ofthe end user (e.g., whether the end user is a system administrator, amanager, or a chief information officer).

In one embodiment, a causal relationship between two events (e.g., twofailures occurring within a networked computing environment) maycomprise a relationship between a first event and a second event,wherein the second event is a consequence of the first event. The secondevent may comprise a direct or indirect consequence of the first event.In another embodiment, a causal relationship between two events maycomprise a relationship between a first event and a second event,wherein if the first event occurs, then the second event must alsooccur. However, if the second event occurs, it does not necessarily meanthat the first event has also occurred. As an example, if electricalpower to a server running an application is disconnected from the server(i.e., a first event), then the application running on the server willbe terminated (i.e., if the first event occurs, then the second event ofthe application being terminated will also occur). However, if anapplication running on a server terminates unexpectedly, then thetermination may be due to other causes besides the electrical powerbeing disconnected from the server.

In some cases, a root cause identification tool may aggregate data froma plurality of information technology management software tools thatmonitor different layers of an IT infrastructure in order to identify aroot cause for a performance or availability issue affecting the ITinfrastructure. In one example, a service (or software service) providedby an IT infrastructure (e.g., an online social networking servicerequiring access to a database application and a messaging applicationto provide the social networking service) may require the availabilityof a plurality of applications. The plurality of applications may run ona plurality of servers located in different locations around the world.In one example, a first set of the plurality of servers may be locatedin a first country and a second set of the plurality of servers may belocated in a second country. The first set of servers may be locatedwithin a room (or within multiple rooms spanning multiple buildings) andmounted on a plurality of racks (e.g., the room may include five serverracks, with each of the five server racks holding 20 servers). The firstset of servers may be interconnected via a first network, the second setof servers may be interconnected via a second network, and the firstnetwork may be connected to the second network via a third network(e.g., the Internet or an extranet).

The root cause identification software tool (or application) mayidentify a root cause of a performance or availability issue affectingthe IT infrastructure upon receiving an alert message from aservice-level management application of the plurality of informationtechnology management software tools monitoring the IT infrastructure.In one example, in the event that a cooling system (e.g., a fan or anair conditioning unit) for regulating the temperature of a room (or aportion of the room) storing the first set of servers malfunctions, thenthe ambient temperature of the room may rise causing the first set ofservers to overheat, which in turn causes the first set of servers toshutdown, which in turn causes the applications running on the first setof servers to be terminated, which in turn may lead to a performanceissue for the service (e.g., a slow down for operations performed viathe online social networking service) or an availability issue for theservice (e.g., users may not be able to use or access the online socialnetworking service). In this case, the plurality of informationtechnology management software tools may overwhelm system administratorswith numerous alarms regarding the unexpected termination ofservice-critical applications and the unexpected shutdown of numerousservers within the room.

In order to quickly diagnosis the root cause of the problems and toprovide a quick resolution to the performance and availability issues,the root cause identification software tool may aggregate data from theplurality of information technology management software tools (e.g.,acquiring updated information every minute), identify a service-relatedperformance or availability issue, identify causal relationshipsassociated the service-related performance or availability issue basedon the aggregated data, determine a root cause of the service-relatedperformance or availability issue based on the causal relationships, andreport the root cause and possible remedies to the service-relatedperformance or availability issue to an end user of the root causeidentification tool based on a role of the end user. The end user mayreceive a single message or an alarm regarding the cooling system withinthe room being the root cause of the performance or availability issue.The root cause identification software tool may also suppress allwarnings or alarms generated from the plurality of informationtechnology management software tools that are a consequence of the rootcause (e.g., that are a consequence of the cooling system malfunction).Thus, one benefit of automatically identifying the root cause of theperformance or availability issue may be a quicker resolution of theissue and reduced downtime for the service.

FIG. 1 depicts one embodiment of a networked computing environment 100in which the disclosed technology may be practiced. Networked computingenvironment 100 includes a plurality of computing devices interconnectedthrough one or more networks 180. The one or more networks 180 allowcomputing devices and/or storage devices to connect to and communicatewith other computing devices and/or other storage devices. As depicted,a mobile device 120, a mobile device 130, a mobile device 140, and adata center 150 are in communication with each other via one or morenetworks 180. The data center 150 includes a server 160 (e.g., an edgeserver) in communication with a server 158 (e.g., an application server)that is in communication with a storage device 155 (e.g., anetwork-attached storage device). In some cases, the storage device 155may comprise a hard disk drive, a magnetic tape drive, or a solid-statedrive. In some embodiments, the networked computing environment mayinclude other computing and/or storage devices not shown. For example, adata center may include thousands of servers and/or data storage devicesin communication with each other. The data storage devices may comprisea tiered data storage infrastructure (or a portion of a tiered datastorage infrastructure). In some cases, a tiered data storageinfrastructure may include redundant arrays of independent disks and/orstorage area networks.

The one or more networks 180 may include a secure network such as anenterprise private network, an unsecure network such as a wireless opennetwork, a local area network (LAN), a wide area network (WAN), and theInternet. Each network of the one or more networks 180 may include hubs,bridges, routers, switches, and wired transmission media such as a wirednetwork or direct-wired connection. The one or more networks 180 mayinclude an extranet or other private network for securely sharinginformation or providing controlled access to applications.

A server, such as server 160, may allow a client to download information(e.g., text, audio, image, and video files) from the server or toperform a search query related to particular information stored on theserver. In some cases, server 160 may act as a mail server or a fileserver. In general, a “server” may include a hardware device that actsas the host in a client-server relationship or a software process thatshares a resource with or performs work for one or more clients.Communication between computing devices in a client-server relationshipmay be initiated by a client sending a request to the server asking foraccess to a particular resource or for particular work to be performed.The server may subsequently perform the actions requested and send aresponse back to the client.

One embodiment of server 160 includes a network interface 165, processor166, and memory 167, all in communication with each other. Networkinterface 165 allows server 160 to connect to one or more networks 180.Network interface 165 may include a wireless network interface, a modem,and/or a wired network interface. Processor 166 allows server 160 toexecute computer readable instructions stored in memory 167 in order toperform processes discussed herein. In some cases, the server 160 mayestablish a secure connection with one or more computing devices (e.g.,using a virtual private network connection). Processor 166 may compriseone or more processing elements (e.g., multiple CPUs). In oneembodiment, server 160 may store data in a database 169 or acquire datato be processed from the database 169. The stored data associated withdatabase 169 may reside in memory 167.

The networked computing environment 100 may provide a cloud computingenvironment for one or more computing devices. Cloud computing refers toInternet-based computing, wherein shared resources, software, and/orinformation are provided to one or more computing devices on-demand viathe Internet (or other global network). The term “cloud” is used as ametaphor for the Internet, based on the cloud drawings used in computernetworking diagrams to depict the Internet as an abstraction of theunderlying infrastructure it represents. The networked computingenvironment may comprise a cloud computing environment providingSoftware-as-a-Service (SaaS) or Infrastructure-as-a-Service (Iaas)services. Saas may refer to a software distribution model in whichapplications are hosted by a service provider and made available to endusers over the Internet. The networked computing environment 100 mayprovide a software service that requires the availability of one or moreapplications to be operational.

The networked computing environment 100 may comprise an ITinfrastructure for providing one or more services. In some embodiments,a root cause identification tool (or software application) may run on aserver, such as server 158, while various services are provided by thenetworked computing environment 100. The root cause identification toolmay aggregate data from a plurality of information technology managementsoftware tools that monitor different layers of the IT infrastructure inorder to identify a root cause for a performance or availability issueaffecting the IT infrastructure. In one example, a service (or softwareservice) provided by the IT infrastructure (e.g., an online socialnetworking service requiring access to a database application and amessaging application to provide the social networking service) mayrequire the availability of a plurality of applications. The root causeidentification tool may detect a performance or availability issueaffecting one of the plurality of applications, identify causalrelationships associated with the performance or availability issuebased on the aggregated data, determine a root cause of the performanceor availability issue based on the causal relationships, and report theroot cause and possible remedies to the performance or availabilityissue to an end user of the root cause identification tool based on arole of the end user.

FIG. 2 depicts one embodiment of a portion of a data center, such as aportion of data center 150 in FIG. 1. The portion of a data center 250includes a server rack 210 holding servers 212-214, a power unit 220 forpowering the servers 212-214, and a temperature unit 222 for monitoringand regulating an ambient temperature surrounding the servers 212-214.The server rack 210 may be placed within a first part of a room storingthe data center. The portion of a data center 250 includes a server rack230 holding servers 232-234, a power unit 240 for powering the servers232-234, and a temperature unit 242 for monitoring and regulating anambient temperature surrounding the servers 232-234. The server rack 230may be placed within a second part of a room storing the data center.The temperature units may acquire temperature information fromtemperature sensors located outside a server rack, located inside aserver rack, inside a server, or inside components inside the server.For example, a temperature unit may monitor temperatures associated witha room, a server rack, a server box housing a server, or components ofthe server (e.g., a semiconductor chip or a processing core used by theserver).

In one embodiment, a root cause identification tool may acquire locationinformation (e.g., GPS information or a location relative to a map of aroom) associated with server racks, servers, power units, andtemperature units in order to identify root cause failures that arecommon to different computing devices or other components located withina data center. For example, the server rack 210 may be associated with afirst physical location within a room and the server rack 230 may beassociated with a second physical location within the room. A root causeidentification tool may continuously monitor services and/orapplications provided by the servers within the data center. In responseto detecting that an application has unexpectedly failed or has aperformance issue (e.g., response times for the application are greaterthan a threshold), the root cause identification tool may aggregate datafrom a plurality of monitoring applications that monitor differentlayers of the data center infrastructure. The root cause identificationtool may then determine that multiple applications have failedcorresponding with servers within a particular server rack, such asserver rack 210, based on the location information associated with theparticular server rack. The root cause identification tool may determinethat a power unit for providing power to the particular server rack hasfailed and is therefore a root cause for the failure of the multipleapplications. In some cases, if the power unit 220 is also used to powerthe temperature unit 222, then a failure of the power unit 220 may alsocause a failure of the temperature unit 222 and be the root cause foralarms associated with the failure of the temperature unit 222. Inanother example, the root cause identification tool may determine that atemperature unit for regulating the temperature of the particular serverrack has failed and is therefore a root cause for the failure of themultiple applications.

FIG. 3 depicts one embodiment of a monitoring system 350 including aroot cause identification manager 330 for identifying a root cause of aperformance or availability issue affecting a networked computingenvironment. The monitoring system 350 includes a plurality ofmonitoring applications for monitoring different layers of the networkedcomputing environment. The plurality of monitoring applications includesa service-level monitor 302, a network-level monitor 304, anapplication-level monitor 306, a virtualization-level monitor 308, acomputing-level monitor 312, a storage-level monitor 314, a power-levelmonitor 316, and a temperature-level monitor 318. The root causeidentification manager 330 is in communication with each of theplurality of monitoring applications and may aggregate data (e.g.,alarms generated by the monitoring applications, as well as log filesgenerated by devices monitored by the monitoring applications) and storethe aggregated data in a database, such as aggregated data database 337.

The service-level monitor 302 may monitor the performance andavailability of services provided by the networked computingenvironment. In one embodiment, the service-level monitor 302 maymonitor response times experienced by various end users of a service.The network-level monitor 304 may monitor the performance of networksassociated with the networked computing environment. In one embodiment,the network-level monitor 304 may monitor the performance of a networkconnecting an end user device (e.g., a mobile device or workstation)with an application server. The network-level monitor 304 may monitorresponse times and availability of devices connected to the network, aswell as track bandwidth utilization and latency across the variouscomponents of the network. The application-level monitor 306 may monitorthe performance of applications provided by the networked computingenvironment. In one embodiment, the application-level monitor 306 maymonitor a status history associated with a particular application (e.g.,whether the particular application is running, halted, ornon-responsive), the volume of transactions processed by the particularapplication over time, the response times of the particular applicationover various load conditions (e.g., average response times under a peakload), and the computing and storage resources consumed by theparticular application over time (e.g., the memory footprint). Thevirtualization-level monitor 308 may monitor the performance of avirtual machine running a particular application.

The computing-level monitor 312 may monitor the performance of acomputing device running a particular application. In one embodiment,the computing-level monitor 312 may monitor CPU performance, CPU coreperformance, and physical memory usage for the computing device. Thestorage-level monitor 314 may monitor the performance of a storagedevice connected to a network. In one embodiment, the storage-levelmonitor 314 may monitor disk usage and disk access times for the storagedevice over time. The power-level monitor 316 may monitor whether poweris connected to a particular device (e.g., electrical power is connectedto a particular computing device or a particular storage device). Thepower-level monitor 316 may also monitor whether power is connected tospecific servers, storage devices, switches, and other data centercomponents. The temperature-level monitor 318 may monitor temperaturesassociated with a room, a server rack, physical servers, and storagedevices. The temperature-level monitor 318 may also generate andtransmit alarms if a particular temperature is not within a desiredrange or if a temperature regulator is not able to regulate atemperature associated with a hardware component of the networkedcomputing environment to be within the desired range. In one example,the temperature-level monitor 318 may generate an alarm if a temperaturewithin a data center room is not between 10 degrees Celsius and 30degrees Celsius. In another example, the temperature-level monitor 318may generate an alarm if a processor core associated with a particularserver has a chip temperature that is greater than 105 degrees Celsius.

The root cause identification manager 330 includes an alarm aggregator331, an alarm relationships identifier 332, a causal chain identifier334, a report generator 336, and an aggregated data database 337. Theroot cause identification manager 330 may comprise an applicationrunning on a server, such as server 158 in FIG. 1. The alarm aggregator331 acquires data from the plurality of monitoring applications andaggregates the data. The aggregated data may include alarms or alertsgenerated by the plurality of monitoring applications, as well as logfiles generated by devices being monitored by the plurality ofmonitoring applications. The alarm aggregator 331 may aggregate dataupon detection of a particular event (e.g., upon detection of aservice-level failure) or may aggregate data periodically (e.g., every 5minutes) and store the aggregated data in the aggregated data database337. The alarm relationships identifier 332 may map a plurality ofalarms generated by the plurality of monitoring applications to nodes ina failure graph. Each alarm of the plurality of alarms may be associatedwith a description of the underlying failure detected and a timestampfor when the underlying failure occurred.

In one embodiment, each node in a failure graph may correspond with anindividual alarm of the plurality of alarms or a collection of alarms ofthe plurality of alarms. In another embodiment, each node in a failuregraph may correspond with a particular type of alarm at a particularlevel in a networked computing environment hierarchy (e.g., CPUutilization alarms associated with a particular server or applicationperformance alarms associated with a particular application). In somecases, a failure graph may be generated for each alarm of the pluralityof alarms generated by the service-level monitor 302.

The alarm relationships identifier 332 may generate directed edgeswithin a failure graph corresponding with a causal relationship betweenpairs of nodes in the failure graph. A directed edge in the failuregraph may represent that a first failure is a direct consequence ofanother failure. For example, the first failure may correspond with afirst node in the failure graph with a directed edge to a second node inthe failure graph corresponding with a second failure that is a directconsequence of the first failure. In this case, the directed edgerepresents a causal relationship between the first failure and thesecond failure. In one embodiment, the failure graph may comprise adirected acyclic graph. In another embodiment, the failure graph maycomprise a Bayesian network with causal relationship probabilitiesassigned to each of the directed edges. The causal relationshipprobabilities may be stored in tables linked to the edges of the failuregraph. In this case, the structure of the failure graph and the assignedprobabilities may be learned from the aggregated data. In one example,the graph structure of the Bayesian network may be determined usingmachine learning techniques based on the aggregated data and changes inthe aggregated data over time (e.g., the aggregated data stored in theaggregated data database 337 may be used as training data for learningthe causal relationships between the nodes over time).

The causal chain identifier 334 may identify a chain of failuresbeginning from a first node in the failure graph (e.g., a leaf node) andending at a root node of the failure graph. The root node of the failuregraph may correspond with the highest-level alarm generated by theplurality of monitoring applications. The leaf nodes of the failuregraph may correspond with root causes of the highest-level alarm. Theleaf nodes may comprise nodes without any predecessor nodes or nodeswithout any incoming directed edges from another node in the failuregraph. A chain of failures may comprise a set of nodes along a path froma leaf node in the failure graph to the root node of the failure graph.

In one embodiment, the causal chain identifier 334 may identify aparticular chain of failures within the failure graph based on anestimated time to fix a failure associated with the leaf node of theparticular chain of failures. In another embodiment, the causal chainidentifier 334 may identify a particular chain of failures within thefailure graph based on a length of the particular chain of failures. Forexample, the particular chain of failures may comprise the shortestchain of failures in the failure graph. In another embodiment, thecausal chain identifier 334 may identify a particular chain of failuresbased on a number of alarms that are a consequence of the leaf node ofthe particular chain of failures. For example, the particular chain offailures may include a leaf node in which fixing the failure associatedwith the leaf node will fix the greatest number of unresolved alarms.

The report generator 336 may generate and transmit a report to a targetrecipient of the report based on an identified chain of failures. In oneembodiment, the report generator 336 may identify a role associated witha target recipient and output a report to the target recipient based onthe role and an ontology defined for the role. The role of the targetrecipient may be determined based on a user name, an employeeidentification number, or an email address associated with the targetrecipient. In one example, a person with a technical role within anorganization may receive a report with technical information (e.g.,server utilization information), while a person with a non-technicalrole within the organization may receive a report with business-focusedinformation (e.g., the number of people who can currently connect to aparticular application or the estimated downtime for the particularapplication).

In some embodiments, an application-level monitoring application, suchas application-level monitor 306, may generate a first alarm based on anerror coming from an application server (e.g., that a databaseapplication is not responsive or that the application server cannotconnect to the database). In response to the first alarm, the root causeidentification manager 330 may identify causally related alarmsgenerated from other monitoring applications (e.g., the network-levelmonitor 304 and other monitoring applications monitoring a networkedcomputing environment). In one example, close to the time at which thefirst alarm from the application-level monitoring application wasgenerated, the root cause identification manager 330 may identify asecond alarm generated from the network-level monitor 304 related to aparticular network switch malfunctioning and determine whether the firstalarm is a direct consequence of the second alarm (i.e., that thefailure of the particular network switch would cause the error comingfrom the application server). In some cases, the failure of theparticular network switch may be caused by a hardware-related issue(e.g., due to a power failure or a physical connection issue) orsoftware-related issue (e.g., the particular network switch wasoverloaded with too many connections). If the first alarm is determinedto be a direct consequence of the second alarm, then the root causeidentification manager 330 may attempt to identify a third alarm that isthe cause of the second alarm. If a third alarm is not identified, thenthe second alarm may be outputted as the root cause of the error comingfrom the application server.

In some embodiments, a root cause identification tool, such as rootcause identification manager 330, may aggregate data from a plurality ofinformation technology management software tools periodically or inresponse to a service-level performance issue being detected (e.g., aservice is no longer available to an end user of the service). Theaggregated data may include service-level data related to a serviceprovided by a networked computing environment, such as the availabilityof the service and response times associated with the service. Theservice may require a plurality of applications to be available (e.g.,an online personal information manager may require a word processingapplication, an email application, and a database application to beavailable). The aggregated data may include application-level datarelated to the plurality of applications, such as a status of each ofthe plurality of applications (e.g., currently running, halted, orterminated) and an identification of a first set of servers which arerunning the plurality of applications. The aggregated data may includenetworking-level data associated with networks connected to the firstset of servers, such as the resources available in the network andnetwork utilization metrics. The aggregated data may includevirtualization-level data associated with the performance of virtualmachines on which applications are running. The aggregated data mayinclude device-level data associated with device level performancemetrics (e.g., computing device utilization or storage deviceutilization). From the aggregated data corresponding with the differentperspectives offered by the plurality of information technologymanagement software tools, the root cause identification tool maydetermine causal relationships between failures occurring at differentlayers within a failure hierarchy (e.g., represented as directed edgesbetween failure nodes in a directed acyclic graph) and identify a rootcause of a service-level performance issue based on the causalrelationships. In one example, a root cause of a service-relatedperformance issue may comprise a disk failure or a power failure to adata storage unit storing a database critical to a service.

FIG. 4 depicts one embodiment of an infrastructure mapping 450 formapping services or applications to components of a networked computingenvironment that support the services or applications at a particulartime. The particular time may correspond with a time of failure or atime prior to the time of failure. The infrastructure mapping maycomprise a directed graph in which an application is mapped to hardwareand software components of the networked computing environment thatsupport the execution of the application. As depicted, service 402 mayrequire one or more applications to be available including application404 at a particular time. The application 404 may be running on avirtual machine 406 at the particular time. The virtual machine 406 maybe running on a server 408 at the particular time. The server 408 mayrely on operating system 418 to run the application 404. To run theapplication 404, the server 408 may require network connections 416 andaccess to storage 414 at the particular time. To run the application404, server 408 may require a particular temperature to be regulatedusing temperature system 410. The particular temperature may correspondwith a die temperature, a server temperature, a server rack temperature,or an ambient room temperature. The temperature system 410 may require apower connection via a power system 412 to be operational.

In one embodiment, an infrastructure mapping may be used by a root causeidentification tool, such as root cause identification manager 330 inFIG. 3, to determine which server an application was running on when theapplication failed. The identification of the server may be used foridentifying alarms that may have a relationship to the failure of theapplication. If the root cause identification tool determines that theserver had a failure, then the infrastructure mapping may be used toidentify a power system responsible for powering a cooling systemassociated with the server or for powering the server itself.

FIG. 5A depicts one embodiment of a failure graph template 550. FIG. 5Bdepicts one embodiment of a failure graph 560 generated using thefailure graph template 550 of FIG. 5A.

The failure graph template 550 includes a service failure node 502 thatis the root node of the failure graph template 550, an applicationfailure node 506 with a directed edge towards the service failure node502, a network connection failure to application server node 504 with adirected edge towards the service failure node 502, a server failurenode 510 with a first directed edge to the application failure node 506and a second directed edge to a virtual machine failure node 512, adatabase failure node 508 with a directed edge to the applicationfailure node 506. The virtual machine failure node 512 has a directededge to the application failure node 506. The failure graph template 550includes a power failure node 514 with a directed edge to the serverfailure node 510 and an overheating failure node 516 with a directededge to the server failure node 510.

A failure graph template may include a root node, such as servicefailure node 502, and one or more predecessor notes. The root node maycomprise a node without any directed edges from the node to anothernode. The root node may correspond with a failure within a networkedcomputing environment for which a root cause is sought. In oneembodiment, each node in a failure graph template may correspond with anindividual alarm or a collection of alarms acquired from a plurality ofmonitoring applications monitoring the networked computing environment.For example, each node in a failure graph template may correspond with aparticular type of alarm at a particular level in a networked computingenvironment (e.g., CPU utilization alarms associated with a particularserver or application performance alarms associated with a particularapplication).

In some embodiments, a failure graph template may comprise a directedacyclic graph. In another embodiment, a failure graph template maycomprise a Bayesian network with causal relationship probabilitiesassigned to each of the directed edges. The causal relationshipprobabilities may be stored in tables linked to the edges of the failuregraph template. In this case, the structure of the failure graphtemplate and the assigned probabilities may be learned from dataaggregated from a plurality of monitoring applications monitoring anetworked computing environment. In one example, the graph structure ofthe Bayesian network may be determined using machine learning techniquesbased on the aggregated data and changes in the aggregated data overtime (e.g., the aggregated data may be used as training data forlearning the causal relationships between the nodes over time).

In some embodiments, structural equation models may be used to determinethe causal relationships associated with the directed edges in a failuregraph. In one example, a structural equation model (SEM) may berepresented using a causal graph (or a causal structure). The causalgraph may comprise a directed acyclic graph (DAG) in which the verticesof the DAG correspond with variables and the edges of the DAG correspondwith whether there is a direct causal relationship between two of thevariables. If a directed edge runs from a first node of the DAG to asecond node of the DAG, then the first node may be referred to as aparent node of the second node (i.e., an ancestor) and the second nodemay be referred to as a child node of the first node (i.e., adescendant). In another example, a SEM may be represented using a set ofequations, wherein each equation of the set of equations describes avariable in terms of other variables which are its direct causes.

In some cases, the causal relationships may be determined by taking intoaccount both observational data (e.g., aggregated data and the changesto the aggregated data over time) and interventional data. Anintervention may comprise a controlled experiment, wherein values areassigned to one or more variables. In some cases, interventions may beused to evaluate causal counterfactual conditionals and to answercounterfactual questions, such as what other failures in a networkedcomputing environment would occur if a particular failure occurredassociated with an intervention. The interventions may be applied to atest environment that mirrors a production-level networked computingenvironment. After the interventions have been performed, theconsequences of the interventions may be observed to identify the causalrelationships. Thus, causal relationships may be determined by bothpassively observing data (e.g., monitoring the alarms naturallyoccurring within a networked computing environment over a particularperiod of time) and by actively forcing failures to occur to a testenvironment that mirrors the networked computing environment.

In some embodiments, the SEM that best fits both the observational dataand the interventional data may be determined using a constraint-basedlearning approach and/or a search-and-score-based learning approach. Aconstraint-based learning approach determines a list of invarianceproperties (e.g., conditional independencies) implied in theobservational data and then rules out all causal structures which areincompatible with the list of invariance properties. Asearch-and-score-based learning approach generates a first causalstructure, computes a score for the first causal structure, and theniteratively generates modified causal structures derived from the firstcausal structure, and scores the modified causal structures. Thisprocedure may be repeated over many generations of modified causalstructures in order to determine the causal structure that produces thebest score.

In one embodiment, an initial causal graph may be generated based on thelayers associated with a networked computing environment. For example,alarms corresponding with a computing system layer may be assigned toancestor nodes of alarms corresponding with an application layer. Afterthe initial model has been generated, it may be modified in order tofind the best fit with both the observational data and theinterventional data. In some cases, the constraint-based approach andthe search-and-score-based approach may be combined to form a hybridapproach. More information regarding the use of interventions fordetermining causal relationships and for using a calculus ofcounterfactuals may be found in Pearl, Judea. Causality: Models,Reasoning, and Inference (Cambridge University Press, 2009).

As depicted in FIG. 5A, the failure graph template 550 may representcausal relationships between potential failures for the service failurenode 502. The service failure node 502 may correspond with an alarmgenerated from a service-level monitor, such as service-level monitor302 in FIG. 3. In one example, the service failure node 502 maycorrespond with the unavailability of a software service. The failuregraph template 550 may include directed edges corresponding with causalrelationships between pairs of nodes in the failure graph template. Adirected edge in the failure graph template may represent that a firstfailure is a direct consequence of another failure. For example, thefirst failure may correspond with a first node in the failure graphtemplate with a directed edge to a second node in the failure graphtemplate corresponding with a second failure that is a directconsequence of the first failure. In this case, the directed edgerepresents a causal relationship between the first failure and thesecond failure. In one embodiment, the failure graph template 550 may beused by a root cause identification tool, such as root causeidentification manager 330 in FIG. 3, to determine causal relationshipsbetween failures occurring within a networked computing environment andto generate a failure graph, such as failure graph 560 in FIG. 5B.

Referring to FIG. 5B, upon detection of an alarm regarding serviceresponse time 572 from a service-level monitor, the root causeidentification tool may map the alarm regarding service response time572 to a root node of a failure graph template. The failure graphtemplate may be one of a plurality of failure graph templates. Each ofthe plurality of failure graph templates may correspond with differenttypes of failures occurring within a networked computing environment. Inthis case, the alarm regarding service response time 572 is mapped tothe root node of failure graph template 550 in FIG. 5A. Using thefailure graph template 550, the root cause identification tool may thenlook for alarms associated with network connection failures to anapplication server running an application. The application with a slowresponse time and the application server with which to find networkconnection failures may be identified using an infrastructure mappingfor the service at the time that the service response time issueoccurred, such as infrastructure mapping 450 in FIG. 4.

Once the application causing the service response time issue and theapplication server running the application have been identified, theroot cause identification tool may analyze the alarms generated from anetwork-level monitor for potential network performance issues. In thiscase, no alarms were found regarding any network performance issues withthe network connection to the application server. Next, the root causeidentification tool may then analyze alarms associated with theapplication. In this case, an alarm regarding application response time576 is found and mapped to a node corresponding with the applicationfailure node 506 in FIG. 5A. After the application failure has beenfound, a server running the application with the response time issue maybe identified using the infrastructure mapping. As a virtual machine wasnot used for running the application, alarms associated with a virtualmachine failure did not need to be considered by the root causeidentification tool. Once the server running the application has beenidentified, the root cause identification tool may analyze the alarmsgenerated from a system-level monitor for server-related performanceissues. In this case, an alarm regarding server failure 580 is found andmapped to a node corresponding with the server failure node 510 in FIG.5A. After the server failure has been found, the root causeidentification tool may then check for overheat failures and powerfailures potentially causing the server failure. In this case, no alarmsregarding overheating were found and an alarm regarding power failure584 is found and mapped to a node corresponding with the power failurenode 514 in FIG. 5A.

As depicted in FIG. 5B, the failure graph 560 includes a leaf nodecorresponding with the alarm regarding power failure 584 and a root nodecorresponding with the alarm regarding service response time 572. A pathfrom the leaf node of the failure graph to the root node of the failuregraph including the nodes 584, 580, 576, and 572 comprises a chain offailures. Although only a single chain of failures is depicted in FIG.5B, a failure graph may include more than one failure chain. In caseswere multiple failure chains exist, a particular chain of failureswithin the failure graph may be determined based on an estimated time tofix a failure associated with the leaf node of the particular chain offailures. In other cases, a particular chain of failures within thefailure graph may be determined based on a length of the particularchain of failures (e.g., the chain of failures with the fewest number ofnodes in the path).

FIG. 6A is a flowchart describing one embodiment of a process foridentifying a root cause of a failure or performance-related issuewithin a networked computing environment. In one embodiment, the processof FIG. 6A is performed by a server, such as server 158 in FIG. 1.

In step 602, an alarm corresponding with a service-level issue affectinga networked computing environment is detected. The service-level issuemay correspond with a service provided by the networked computingenvironment. The alarm may be associated with a time of failure (e.g.,using a timestamp). The service-level issue may correspond with theunavailability or non-responsiveness of one or more applicationsrequired by a service. In step 604, one or more applications associatedwith the service-level issue are identified. The one or moreapplications may be identified using an infrastructure mapping for theservice at the time of failure, such as infrastructure mapping 450 inFIG. 4. In step 606, an infrastructure mapping for a first applicationof the one or more applications is acquired. The infrastructure mappingmaps the first application to infrastructure (or components of thenetworked computing environment) that supported the first application atthe time of failure. In step 608, a plurality of alarms from a pluralityof performance management tools for monitoring the networked computingenvironment is aggregated. The plurality of alarms may be stored in anaggregated data database, such as aggregated data database 337 in FIG.3.

In step 610, a failure graph is generated based on the infrastructuremapping and the plurality of alarms. One embodiment of a process forgenerating a failure graph is described later in reference to FIG. 6C.In step 612, a chain of failures is determined based on the failuregraph. The chain of failures may include a leaf node of the failuregraph and the root node of the failure graph. In step 614, a root causeof the service-level issue is identified based on the chain of failures.In step 616, the root cause of the service-level issue is outputted. Inone embodiment, the root cause of the service-level issue is transmittedto a target recipient based on a role associated with the targetrecipient. In another embodiment, a help desk ticket is automaticallygenerated for fixing a failure associated with the leaf node of thechain of failures.

In some embodiments, a software service provided by a networkedcomputing environment may experience a service-related performance oravailability issue (e.g., the software service may not be available toend users or the response time for the software service to the end usersmay be more than a threshold amount of time). In response to detectingthe service-related issue affecting the service (e.g., an alarm isreceived regarding unavailability of the service), a root causeidentification tool may aggregate data from a plurality of informationtechnology management software tools monitoring the networked computingenvironment, identify causal relationships between a plurality offailures associated with the service-related issue based on theaggregated data, determine a chain of failures of the plurality offailures based on the causal relationships, identify a root cause of theservice-related issue based on the chain of failures, and transmit analarm corresponding with the root cause (e.g., sending a text message toa system administrator responsible for maintenance of a hardware deviceassociated with the root cause).

The aggregated data may include alarms from various monitoringapplications monitoring the networked computing environment, such as anapplication performance management tool and a network performancemanagement tool, as well as log files generated by devices within thenetworked computing environment. Each of the alarms may correspond witha possible failure associated with the service-related issue. The alarmsmay be classified as failures correlated with the service-related issueand mapped to nodes in a failure graph, wherein each failure correspondswith a node in the failure graph and directed edges in the failure graphdetermine whether a failure is a direct consequence of another failurein the failure graph. For example, a first failure may correspond with afirst node in the failure graph with a directed edge to a second node inthe failure graph corresponding with a second failure that is a directconsequence of the first failure. In this case, the directed edgerepresents a causal relationship between the first failure and thesecond failure. The root node of the failure graph may correspond withthe highest-level failure (e.g., the service-related issue) in thefailure graph. The leaf nodes in the failure graph may correspond withroot causes of the highest-level failure. The leaf nodes may comprisenodes without any predecessor nodes or nodes without any incomingdirected edges from another node. A chain of failures may comprise apath from a leaf node in the failure graph to the root node of thefailure graph. In some embodiments, the failure graph may comprise adirected acyclic graph.

In some cases, the alarms associated with failures occurring atdifferent layers within the networked computing environment may bemapped to different nodes in the failure graph. In one embodiment, thenodes of the failure graph may include a service layer node (e.g., aperformance-related issue associated with providing a payment processingservice), a network layer node (e.g., associated with networking issuesrelated to providing network connections to end users of the paymentprocessing system), an application layer node (e.g., associated withapplication issues for applications required to provide the paymentprocessing system), an access control layer node (e.g., associated withdata access issues for an application), a virtualization layer node(e.g., associated with virtual machine issues), a computing systemslayer node (e.g., associated with physical processor hardware issues), astorage systems layer node (e.g., associated with physical storagehardware issues), a temperature control layer node (e.g., associatedwith temperature regulation issues affecting the computing and storagehardware), and a power delivery layer node (e.g., associated with powerdelivery issues to the computing and storage hardware). In anotherembodiment, a networked computing environment may be divided intolayers, with each layer corresponding with a different class of entities(e.g., processors, applications, networks, power supplies, switches,etc.). The nodes of the failure graph may then correspond with thedifferent classes of entities within the networked computingenvironment.

FIG. 6B is a flowchart describing an alternative embodiment of a processfor identifying a root cause of a failure or performance-related issuewithin a networked computing environment. In one embodiment, the processof FIG. 6B is performed by a server, such as server 158 in FIG. 1.

In step 622, an alarm corresponding with a failure in a networkedcomputing environment is detected. The alarm may be associated with atime of failure. The alarm may be acquired from an application-levelmonitor, such as application-level monitor 306 in FIG. 3. In step 624,one or more applications associated with the failure are identified. Inone embodiment, the one or more applications may be identified by firstidentifying a service associated with the failure and then determiningthe one or more applications supporting the service at the time offailure. In step 626, an infrastructure mapping for a first applicationof the one or more applications is acquired. The infrastructure mappingmaps the first application to infrastructure (e.g., various componentswithin the networked computing environment) that supported the firstapplication at the time of failure.

In step 628, a plurality of alarms from a plurality of performancemanagement tools monitoring the networked computing environment at thetime of failure is aggregated. The plurality of alarms may be stored inan aggregated data database, such as aggregated data database 337 inFIG. 3. In step 630, a failure graph is generated based on theinfrastructure mapping and the plurality of alarms. The alarm maycorrespond with a root node of the failure graph. One embodiment of aprocess for generating a failure graph is described later in referenceto FIG. 6C.

In step 632, a set of leaf nodes of the failure graph is identified. Instep 634, a first leaf node of the set of leaf nodes is identified. Thefirst leaf node may correspond with a root cause of the failure in thenetworked computing environment. In step 636, a chain of failurescorresponding with the first leaf node and the root node of the failuregraph is determined. In step 638, an alert corresponding with the chainof failures is outputted.

In one embodiment, the chain of failures is determined based on a lengthof the chain of failures. For example, the chain of failures maycomprise the shortest chain of failures in the failure graph (i.e., achain of failures with the fewest number of nodes). In anotherembodiment, the chain of failures may be determined based on a number ofalarms that are a consequence of the leaf node of the chain of failures.In one example, the chain of failures may include a leaf node in whichfixing the failure associated with the leaf node will fix the greatestnumber of unresolved alarms of the plurality of alarms. As an example,fixing a leaf node failure associated with fixing a power supply issueaffecting a server rack which in turn affects tens of servers andthousands of applications may be given priority over fixing a memoryutilization issue affecting only a single server. In another embodiment,the chain of failures may be determined based on an estimated time tofix a failure associated with the leaf node of the chain of failures.

FIG. 6C is a flowchart describing one embodiment of a process forgenerating a failure graph. The process described in FIG. 6C is oneexample of a process for implementing step 610 in FIG. 6A or forimplementing step 630 in FIG. 6B. In one embodiment, the process of FIG.6C is performed by a server, such as server 158 in FIG. 1.

In step 672, a plurality of failure layers associated with a networkedcomputing environment is determined. The plurality of failure layers mayinclude a first layer and a second layer. In one embodiment, theplurality of failure layers may include a service layer (e.g., coveringperformance-related issues associated with providing a paymentprocessing service), a network layer (e.g., covering networking issuesrelated to providing network connections to end users of the paymentprocessing system), an application layer (e.g., covering applicationissues for applications required to provide the payment processingsystem), a virtualization layer (e.g., covering virtual machine issues),a computing systems layer (e.g., covering physical processor hardwareissues), a storage systems layer (e.g., covering physical storagehardware issues), a temperature control layer (e.g., coveringtemperature regulation issues affecting the computing and storagehardware), and a power delivery layer (e.g., covering power deliveryissues to the computing and storage hardware). In another embodiment,the plurality of layers may correspond with different classes ofentities associated with the networked computing environment (e.g.,processors, applications, networks, power supplies, switches, etc.).

In step 674, a first application associated with a failure within thenetworked computing environment is identified. If the failure comprisesa system-level failure, then a lookup table of applications running atthe time of the failure supporting the service may be used to identifythe first application. If the failure comprises an application failure,then an alarm associated with the failure acquired from anapplication-level monitor may provide an identification of the firstapplication. In step 676, an infrastructure mapping corresponding withthe first application is acquired. In one embodiment, the infrastructuremapping may provide a mapping of the first application to theinfrastructure or components of the networked computing environment thatsupported the first application at the time of the failure.

In step 678, a plurality of alarms from a plurality of monitoringapplications monitoring the networked computing environment at the timeof the failure is acquired. In step 680, each alarm of the plurality ofalarms is assigned to one of the plurality of failure layers. In oneexample, a first alarm generated by an application-level monitor may bemapped to an application layer of the plurality of failure layers. Instep 682, a first alarm assigned to the first layer is identified. Thefirst alarm may be mapped to a root node of a failure graph. The firstalarm may correspond with a failure within the networked computingenvironment for which a root cause is sought.

In some embodiments, a failure graph template, such as failure graphtemplate 550 in FIG. 5A, may be used to determine causal relationshipsbetween failures occurring within the networked computing environmentand to generate a failure graph, such as failure graph 560 in FIG. 5B.In some embodiments, a failure graph template may comprise a directedacyclic graph representing causal relationships between failuresoccurring within a networked computing environment. In anotherembodiment, a failure graph template may comprise a Bayesian networkwith causal relationship probabilities assigned to each of the directededges. The causal relationship probabilities may be stored in tableslinked to the edges of the failure graph template. In this case, thestructure of the failure graph template and the assigned probabilitiesmay be learned from data aggregated from a plurality of monitoringapplications monitoring a networked computing environment. In oneexample, the graph structure of the Bayesian network may be determinedusing machine learning techniques based on the aggregated data andchanges in the aggregated data over time (e.g., the aggregated data maybe used as training data for learning the causal relationships betweenthe nodes over time).

In step 684, a second set of alarms assigned to the second layer isdetermined based on the infrastructure mapping. In step 686, it isdetermined that the first alarm is a consequence of a second alarm ofthe second set of alarms. In one embodiment, it may be determined thatthe first alarm is a consequence of the second alarm based on thepresence of a directed edge within a corresponding failure graphtemplate. In another embodiment, it may be determined that the firstalarm is a consequence of the second alarm based on aggregated dataacquired from a plurality of monitoring applications monitoring thenetworked computing environment. The aggregated data may include logfile data associated with the plurality of alarms. The aggregated datamay also include help desk ticket data associated with human-enteredhelp desk tickets covering failures and performance issues affecting thenetworked computing environment. In some cases, the help desk ticketsmay be treated as additional alarms to the plurality of alarms. In step688, a failure graph is generated. The failure graph may include a firstnode associated with the first alarm, a second node associated with thesecond alarm, and a directed edge from the second alarm to the firstalarm.

FIG. 7A is a flowchart describing one embodiment of a process forconsolidating multiple alarms generated from a plurality of monitoringapplications monitoring a networked computing environment under a singleroot cause. In one embodiment, the process of FIG. 7A is performed by aserver, such as server 158 in FIG. 1.

In step 702, an alert corresponding with a performance issue in anetworked computing environment is detected. The performance issue maycomprise a failure of a component of the networked computingenvironment. The alert may be acquired from an application-levelmonitor, such as application-level monitor 306 in FIG. 3. In someembodiments, the alert may be generated by the application-level monitorif a current performance metric is outside an acceptable range. As anacceptable range of application performance may vary over time due tovarying conditions, such as server loads, end user usage patterns, dayof the week or month (e.g., weekend days, weekdays, and holidays), timeof day (e.g., during working hours vs. non-working hours), and loadpatterns (e.g., batch mode processing may be performed at a particulartime of day), different baselines of application performance may bedetermined for the varying conditions. In one example, an acceptablerange for application response time may vary based on the time of dayand day of the week.

In step 704, data from a plurality of performance management toolsmonitoring the networked computing environment is aggregated. Theaggregated data may include a plurality of alarms. The aggregated datamay also include log file data associated with the plurality of alarms,as well as help desk ticket data associated with human-entered help desktickets covering failures and performance issues affecting the networkedcomputing environment.

In step 706, a plurality of nodes is generated. Each node of theplurality of nodes may be associated with a different alarm of theplurality of alarms. In step 708, a set of directed edges is determined.Each directed edge of the set of directed edges may correspond with acausal relationship between a pair of the plurality of nodes. In step710, a failure graph is generated. The failure graph may include theplurality of nodes and the set of directed edges. The alert maycorrespond with a root node of the failure graph. One embodiment of aprocess for generating a failure graph was described in reference toFIG. 6C.

In step 712, a first leaf node of the plurality of nodes is identified.The first leaf node may correspond with a root cause of the performanceissue. In step 714, a first chain of failures corresponding with thefirst leaf node and the root node of the failure graph is determined. Instep 716, each alarm of the plurality of alarms that is not associatedwith a node in the first chain of failures is suppressed. In oneexample, each alarm of the plurality of alarms that is not associatedwith the first chain of failures may be hidden or not reported to atarget recipient. In step 718, a consolidated alarm corresponding withthe first chain of failures is outputted. The consolidated alarm maycomprise a report or other message specifying the leaf node of the firstchain of failures. The message specifying a failure associated with theleaf node may be transmitted to a target recipient.

In some embodiments, in response to detecting an alert correspondingwith a service-level failure affecting a service provided by a networkedcomputing environment, a root cause identification tool may aggregate aplurality of alarms from a plurality of performance management toolsmonitoring the networked computing environment. The root causeidentification tool may then generate a failure graph associated withthe service-level failure based on the plurality of alarms, determine afirst set of leaf nodes of the failure graph, determine a first chain offailures based on the first set of leaf nodes, suppress (or hide) alarmsthat are not associated with the first chain of failures, and output aconsolidated alarm associated with the first chain of failures. In oneexample, the consolidated alarm may present information only associatedwith the leaf node of the first chain of failures. In another example,the consolidated alarm may present information only associated with theleaf node of the first chain of failures and the root node of the firstchain of failures.

FIG. 7B is a flowchart describing one embodiment of a process foridentifying a single root cause for multiple failures occurring in anetworked computing environment. In one embodiment, the process of FIG.7B is performed by a server, such as server 158 in FIG. 1.

In step 732, a plurality of alarms associated with a networked computingenvironment is acquired. The plurality of alarms may include a firstalarm and a second alarm. The first alarm and the second alarm maycomprise alarms generated by an application-level monitor, such asapplication-level monitor 306 in FIG. 3. The first alarm may beassociated with a first application being non-responsive and the secondalarm may be associated with a second application being non-responsive.In some embodiments, both the first alarm and the second alarm maycorrespond with two different service-level failures or two differentapplication-level failures occurring within the networked computingenvironment.

In step 734, a first failure graph corresponding with the first alarm isgenerated. In step 736, a second failure graph corresponding with thesecond alarm is generated. In step 738, a first set of leaf nodesassociated with the first failure graph is identified. In step 740, asecond set of leaf nodes associated with the second failure graph isidentified. In step 742, a first leaf node that is common to both thefirst failure graph and the second failure graph is identified. In thiscase, the first set of leaf nodes may include the first leaf node andthe second set of leaf nodes may include the first leaf node. In step744, the first leaf node is outputted. In this case, the first leaf nodemay correspond with a root cause failure that is responsible forgenerating both the first alarm and the second alarm.

In some embodiments, a plurality of failure graphs corresponding withdifferent failures occurring within the networked computing environmentmay be generated and a common root cause for the root nodes of theplurality of failure graphs may be identified. In one embodiment, acommon leaf node (i.e., a common root cause) associated with theplurality of failure graphs may be identified and outputted as a rootcause of the different failures. In one example, two alarmscorresponding with the failures of two different applications may bedetected, two failure graphs may then be generated corresponding withthe two alarms, and a leaf node that is common to both of the twofailure graphs may be identified (e.g., if a server running the twodifferent applications has crashed taking down both applications, thenthe server failure may comprise a common root cause problem for both ofthe application failures). In another example, a common root causefailure leading to multiple alarms may correspond with a single powerfailure (e.g., a power failure to a server rack housing multiple serversrunning numerous applications) or a temperature regulation failure(e.g., a cooling system failure affecting a particular region within adata center).

FIG. 8A is a flowchart describing one embodiment of a process forvisualizing and accessing data associated with causal relationshipsbetween failures occurring within a networked computing environment. Inone embodiment, the process of FIG. 8A is performed by a server, such asserver 158 in FIG. 1.

In step 802, data from a plurality of performance management toolsmonitoring a networked computing environment is aggregated. Theaggregated data may include a plurality of alarms, as well as log filesgenerated by devices within the networked computing environment. In step804, a plurality of nodes is generated. Each node of the plurality ofnodes is associated with a different alarm of the plurality of alarms.In step 806, a set of directed edges is determined. Each directed edgeof the set of directed edges corresponds with a causal relationshipbetween a pair of the plurality of nodes. In step 808, a failure graphis generated. The failure graph may include the plurality of nodes andthe set of directed edges. One embodiment of a process for generating afailure graph was described in reference to FIG. 6C.

In step 810, a first portion of the aggregated data is identified. Thefirst portion supports a causal relationship for a first edge of the setof directed edges. In one embodiment, if the first edge extends from afirst node of the plurality of nodes to a second node of the pluralityof nodes (i.e., the first edge is a directed edge from the first node tothe second node), then the first portion of the aggregated data mayinclude a first alarm associated with the first node and a second alarmassociated with the second node. The first portion of the aggregateddata may also include a log file generated by a device that triggeredthe first alarm.

In step 812, origination information associated with the first portionof the aggregated data is identified. The origination information mayinclude an identification of a first monitoring tool that generated thefirst alarm (e.g., a system-level monitoring tool) and an identificationof a second monitoring tool that generated the second alarm (e.g., anapplication-level monitoring tool). In step 814, a graphicalrepresentation of the failure graph is generated. The graphicalrepresentation may include circles representing nodes of the failuregraph and arrows representing corrected edges of the failure graph. Thegraphical representation may include a first link from the first edge tothe first portion of the aggregated data. The first link may allow anend user to select the first link in order to bring into view the firstportion of the aggregated data or cause the first portion of theaggregated data to be displayed. In step 816, the graphicalrepresentation is displayed. The graphical representation may bedisplayed using a computer monitor or a touch-sensitive display. In step818, the origination data and the first portion of the aggregated datais displayed upon selection of the first link. In one embodiment, theselection of the first link may be made by an end user of a root causeidentification tool.

In some embodiments, a failure graph may be generated in response todetecting a performance issue in a networked computing environment. Thefailure graph may be generated by a root cause identification tool thataggregates data from a plurality of performance management toolsmonitoring the networked computing environment. The aggregated data mayinclude a plurality of alarms spanning multiple layers of the networkedcomputing environment, as well as log file data (e.g., data fromtemporary log files generated by an operating system) generated bydevices within the networked computing environment (e.g., servers). Theroot cause identification tool may identify causal relationships betweenthe plurality of alarms based on the aggregated data and generate afailure graph including directed edges corresponding with the causalrelationships. The root cause identification tool may generate agraphical representation of the failure graph including pointers toportions of the aggregated data supporting the directed edges. In oneexample, each edge of the failure graph may correspond with one or morelinks to portions of the aggregated data supporting the directionalityof the edge (e.g., a link to a log file supporting the causalrelationship). Each node of the failure graph may correspond with a linkto a particular alarm of the plurality of alarms.

In some embodiments, a root cause identification tool may provide a userinterface for accessing the portions of the aggregated data pointed toby the various links within the failure graph. The portions of theaggregated data pointed to by a link may also include ownershipinformation (i.e., who is responsible for the data), originationinformation (i.e., where the data originated from), and physicallocation information (e.g., a physical location of a server within adata center or a physical location of a router or network switch). Theuser interface and graphical representation of the failure graph withlinks to the underlying data provides a quick way to visualize andaccess the data aggregated from the plurality of performance managementtools monitoring the networked computing environment.

In some cases, the user interface may allow an end user of the rootcause identification tool to access portions of a metrics store thatstores the aggregated data and non-structured information, such as logfiles. In this case, the failure graph may be used as an index to accessthe portions of the metrics store relevant to a particular edge or nodein the failure graph. The user interface may allow the end user toquickly look up non-structured information using the failure graph as anindex to the non-structured information.

FIG. 8B is a flowchart describing one embodiment of a process forautomatically generating a report in response to detecting performanceand/or availability issues that occur throughout multiple layers of anetworked computing environment based on a role of a target recipient ofthe report. In one embodiment, the process of FIG. 8B is performed by aserver, such as server 158 in FIG. 1.

In step 842, a first set of alarms is aggregated from a plurality ofperformance management tools monitoring a networked computingenvironment. In step 844, a second set of alarms is aggregated from ahelp desk ticket database. In step 846, a failure graph is generatedbased on the first set of alarms and the second set of alarms. Oneembodiment of a process for generating a failure graph was described inreference to FIG. 6C.

In step 848, a target recipient for a report is identified. In step 850,a plurality of report templates is acquired. Each report template of theplurality of report templates may correspond with a particular employeerole or level of responsibility within an organization. In step 852, afirst report template of the plurality of report templates is determinedbased on a role associated with the target recipient (e.g., the targetrecipient is a system administrator). In step 854, a set of variablesassociated with the first report template is identified. In step 856,data values are assigned to the set of variables using the failure graphgenerated in step 846. In one example, the data values may include anidentification of a root cause failure. In step 858, a first report isgenerated based on the first report template. In step 860, the firstreport is outputted to the target recipient.

In some embodiments, a failure graph may be generated in response todetecting a performance issue in a networked computing environment. Thefailure graph may be generated by a root cause identification tool thataggregates data from a plurality of performance management toolsmonitoring the networked computing environment. The aggregated data mayinclude a plurality of alarms spanning multiple layers of the networkedcomputing environment. The root cause identification tool may identifycausal relationships between the plurality of alarms based on theaggregated data, generate a failure graph including directed edgescorresponding with the causal relationships, and identify a root causeof the performance issue using the failure graph. The root causeidentification tool may identify a role associated with a targetrecipient and output an alert to the target recipient based on the roleand an ontology defined for the role (e.g., if the target recipient is asystem administrator, then a text message regarding the root cause ofthe performance issue may be sent to the system administrator).

In one embodiment, the root cause identification tool may acquire aplurality of report templates, determine a first report template of theplurality of report templates based on a role associated with a targetrecipient, identify a set of variables associated with the first reporttemplate, assign data values to the set of variables using the failuregraph (e.g., the set of variable may be filled in based on a root causeidentified using the failure graph), generate a first report based onthe first report template, and output the first report to the targetrecipient. The role of the target recipient may be determined based on auser name, an employee identification number, or an email addressassociated with the target recipient. Each of the plurality of reporttemplates may be customized such that only information that is mostrelevant to a person with a particular role within an organization isdisplayed or transmitted to the target recipient. For example, a personwith a technical role within an organization may receive an alert withtechnical information (e.g., server utilization information), while aperson with a non-technical role within the organization may receive analert with business-focused information (e.g., the number of people whocan currently connect to a particular application or the estimateddowntime for the particular application).

In one example, a first end user may login to the root causeidentification tool using a user name that is associated with amanagerial role and view a first report corresponding with a firstreport template associated with the managerial role. A second end usermay login to the root cause identification tool using a second user namethat is associated with a system administrator role and view a secondreport corresponding with a second report template associated with thesystem administrator role.

FIG. 9 is a flowchart describing one embodiment of a process forautomatically generating help desk tickets in response to detectingperformance and/or availability issues that occur throughout multiplelayers of a networked computing environment. In one embodiment, theprocess of FIG. 9 is performed by a server, such as server 158 in FIG.1.

In step 902, an alert corresponding with a performance issue in anetworked computing environment is detected. In step 904, data from aplurality of performance management tools monitoring the networkedcomputing environment is aggregated. The aggregated data may include aplurality of alarms, as well as log files generated by devices withinthe networked computing environment. In step 906, a plurality of nodesis generated. Each node of the plurality of nodes is associated with adifferent alarm of the plurality of alarms. In step 908, a set ofdirected edges is determined. Each directed edge of the set of directededges corresponds with a causal relationship between a pair of theplurality of nodes. In step 910, a failure graph is generated. Thefailure graph may include the plurality of nodes and the set of directededges. One embodiment of a process for generating a failure graph wasdescribed in reference to FIG. 6C. The alert detected in step 902 maycorrespond with a root node of the failure graph.

In step 912, a first leaf node of the plurality of nodes is identified.The first leaf node may correspond with a root cause of the performanceissue. In step 914, a first remedy associated with the first leaf nodeis determined. The first remedy may be determined based on a failurelayer associated with the first leaf node. For example, if the firstleaf node is associated with a system-level failure, then the firstremedy may comprise dispatching a server technician or automaticallyrebooting a server. If the first leaf node is associated with anetwork-level failure, then the first remedy may comprise dispatching anetwork technician, limiting external traffic to the network with theperformance issue, or redirecting network traffic from the network withthe performance issue to a different network. In step 916, a help deskticket is automatically generated corresponding with the first leaf nodeand the first remedy in response to detecting the alert in step 902.

In some embodiments, in response to detecting an alert correspondingwith a performance issue affecting a networked computing environment, aroot cause identification tool may aggregate a plurality of alarms froma plurality of performance management tools monitoring the networkedcomputing environment. The root cause identification tool may thengenerate a failure graph associated with the performance issue based onthe plurality of alarms, identify a first leaf node of the plurality ofnodes, determine a first remedy associated with the first leaf node, andgenerate a help desk ticket corresponding with the first leaf node andthe first remedy. In one example, the first leaf node may correspondwith a power failure to a particular server (or server rack) and thefirst remedy may include dispatching a technician to a locationassociated with the particular server. In this case, the automaticallygenerated help desk ticket may cause a technician to be dispatched tofix the particular server (or a power distribution unit on a server racksupporting the particular server) that is the root cause of many alarms.The automatically generated help desk ticket may specify the failureassociated with the first leaf node, the first remedy, a locationassociated with the first remedy, and an estimated time to fix thefailure associated with the first leaf node.

In some embodiments, issue tickets entered into a help desk ticketdatabase may be treated as human generated alarms. For example, a helpdesk ticket may be created specifying that a particular application isnot available. The description provided in the help desk ticket may beparsed for key words and/or the submission of the help desk ticket mayrequire that particular pull-down fields are specified (e.g., regardingthe availability of a particular application or the ability to access aparticular storage device). The root cause identification tool may thenaggregate not only a plurality of alarms from a plurality of performancemanagement tools monitoring the networked computing environment, butalso a second set of alarms corresponding with the issue tickets enteredinto the help desk ticket database. The root cause identification toolmay then aggregate the plurality of alarms and the second set of alarmsand generate a failure graph associated with the performance issue basedon the plurality of alarms and the second set of alarms.

One embodiment comprises a method for monitoring a networked computingenvironment comprising detecting an alert corresponding with aperformance issue within the networked computing environment andaggregating data from a plurality of monitoring applications monitoringthe networked computing environment. The aggregated data includes aplurality of alarms. The method further comprises generating a failuregraph based on the aggregated data. The failure graph includes aplurality of nodes and a set of directed edges. Each directed edge ofthe set of directed edges corresponds with a causal relationship betweena pair of the plurality of nodes. The alert corresponds with a root nodeof the failure graph. The method further comprises identifying a firstleaf node of the plurality of nodes. The first leaf node correspondswith a root cause of the performance issue. The method further comprisesdetermining a first chain of failures corresponding with the first leafnode and the root node of the failure graph and outputting aconsolidated alarm corresponding with the first chain of failures.

One embodiment comprises a system for monitoring a networked computingenvironment comprising a network interface and a processor incommunication with the network interface. The network interface receivesdata from a plurality of monitoring applications monitoring thenetworked computing environment. The processor detects an alertcorresponding with a performance issue within the networked computingenvironment and aggregates the data from the plurality of monitoringapplications. The processor generates a failure graph based on theaggregated data. The failure graph includes a plurality of nodes and aset of directed edges. Each directed edge of the set of directed edgescorresponds with a causal relationship between a pair of the pluralityof nodes. The alert corresponds with a root node of the failure graph.The processor identifies a first leaf node of the plurality of nodes.The first leaf node corresponds with a root cause of the performanceissue. The processor determines a first chain of failures correspondingwith the first leaf node and the root node of the failure graph andoutputs a consolidated alarm corresponding with the first chain offailures.

One embodiment comprises a computer program product comprising acomputer readable storage medium having computer readable program codeembodied therewith. The computer readable program code configured todetect an alert corresponding with a performance issue within anetworked computing environment and to aggregate data from a pluralityof monitoring applications monitoring the networked computingenvironment. The aggregated data includes a plurality of alarms. Thecomputer readable program code configured to generate a failure graphbased on the aggregated data. The failure graph includes a plurality ofnodes and a set of directed edges. Each directed edge of the set ofdirected edges corresponds with a causal relationship between a pair ofthe plurality of nodes. The alert corresponds with a root node of thefailure graph. The computer readable program code configured to identifya first leaf node of the plurality of nodes. The first leaf nodecorresponds with a root cause of the performance issue. The computerreadable program code configured to determine a first chain of failurescorresponding with the first leaf node and the root node of the failuregraph, to suppress each alarm of the plurality of alarms that is notassociated with a node in the first chain of failures, and to output aconsolidated alarm corresponding with the first chain of failures.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousaspects of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularaspects only and is not intended to be limiting of the disclosure. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of anymeans or step plus function elements in the claims below are intended toinclude any disclosed structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present disclosure has been presentedfor purposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of thedisclosure. The aspects of the disclosure herein were chosen anddescribed in order to best explain the principles of the disclosure andthe practical application, and to enable others of ordinary skill in theart to understand the disclosure with various modifications as aresuited to the particular use contemplated.

For purposes of this document, each process associated with thedisclosed technology may be performed continuously and by one or morecomputing devices. Each step in a process may be performed by the sameor different computing devices as those used in other steps, and eachstep need not necessarily be performed by a single computing device.

For purposes of this document, reference in the specification to “anembodiment,” “one embodiment,” “some embodiments,” or “anotherembodiment” may be used to described different embodiments and do notnecessarily refer to the same embodiment.

For purposes of this document, a connection can be a direct connectionor an indirect connection (e.g., via another part).

For purposes of this document, the term “set” of objects, refers to a“set” of one or more of the objects.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. A method for monitoring a networked computingenvironment, comprising: detecting an alert corresponding with aperformance issue within the networked computing environment;aggregating data from a plurality of monitoring applications monitoringthe networked computing environment, the aggregated data comprises aplurality of alarms; generating a failure graph based on the aggregateddata, the failure graph comprises a plurality of nodes and a set ofdirected edges, each directed edge of the set of directed edgescorresponds with a causal relationship between a pair of the pluralityof nodes, the alert corresponds with a root node of the failure graph;detecting a second alert corresponding with a second performance issuewithin the networked computing environment, the alert corresponds with afirst application-level failure of a first application within thenetworked computing environment and the second alert corresponds with asecond application-level failure of a second application within thenetworked computing environment different from the first application;generating a second failure graph based on the aggregated data, thesecond alert corresponds with a root node of the second failure graph;identifying a first set of leaf nodes associated with the failure graph;identifying a second set of leaf nodes associated with the secondfailure graph; and identifying a common leaf node that is common to boththe failure graph and the second failure graph, the first set of leafnodes comprises the common leaf node and the second set of leaf nodescomprises the common leaf node; identifying a first leaf node of theplurality of nodes, the first leaf node corresponds with a root cause ofthe performance issue; determining a first chain of failurescorresponding with the first leaf node and the root node of the failuregraph; and outputting a consolidated alarm corresponding with the firstchain of failures.
 2. The method of claim 1, wherein: the plurality ofmonitoring applications comprises an application-level monitor, anetwork-level monitor, and a system-level monitor.
 3. The method ofclaim 1, wherein: the aggregated data comprises log file data generatedby devices within the networked computing environment.
 4. The method ofclaim 1, wherein: the aggregated data comprises help desk ticketinformation associated with help desk tickets covering performanceissues affecting the networked computing environment.
 5. The method ofclaim 1, further comprising: suppressing each alarm of the plurality ofalarms that is not associated with a node in the first chain offailures.
 6. The method of claim 1, wherein: the outputting aconsolidated alarm corresponding with the first chain of failurescomprises transmitting a message specifying the root cause of theperformance issue to a target recipient.
 7. The method of claim 1,wherein: the outputting a consolidated alarm corresponding with thefirst chain of failures comprises transmitting a message providinginformation only associated with the first chain of failures to a targetrecipient.
 8. The method of claim 1, wherein: the plurality ofmonitoring applications comprises an application-level monitor, theapplication-level monitor generates a first alarm of the plurality ofalarms in response to a performance metric for an application beingoutside of an acceptable range, the acceptable range is determined basedon a time of day.
 9. The method of claim 1, wherein: the performanceissue comprises an unavailability of the first application, thenetworked computing environment comprises a plurality of servers, theroot cause of the performance issue comprises a power failure to a firstserver of the plurality of servers.
 10. A system for monitoring anetworked computing environment, comprising: a network interfaceconfigured to receive data from a plurality of monitoring applicationsmonitoring the networked computing environment; and a processorconfigured to detect an alert corresponding with a performance issuewithin the networked computing environment and aggregate the data fromthe plurality of monitoring applications, the processor configured togenerate a failure graph based on the aggregated data, the failure graphcomprises a plurality of nodes and a set of directed edges, eachdirected edge of the set of directed edges corresponds with a causalrelationship between a pair of the plurality of nodes, the alertcorresponds with a root node of the failure graph, the processorconfigured to detect a second alert corresponding with a secondperformance issue within the networked computing environment, the alertcorresponds with a first application-level failure of a firstapplication within the networked computing environment and the secondalert corresponds with a second application-level failure of a secondapplication within the networked computing environment different fromthe first application, the processor configured to generate a secondfailure graph based on the aggregated data, the second alert correspondswith a root node of the second failure graph, the processor configuredto identify a common leaf node that is common to both the failure graphand the second failure graph, the processor configured to identify afirst leaf node of the plurality of nodes, the first leaf nodecorresponds with a root cause of the performance issue, the processorconfigured to determine a first chain of failures corresponding with thefirst leaf node and the root node of the failure graph and output aconsolidated alarm corresponding with the first chain of failures. 11.The system of claim 10, wherein: the plurality of monitoringapplications comprises an application-level monitor, a network-levelmonitor, and a system-level monitor.
 12. The system of claim 10,wherein: the aggregated data comprises a plurality of alarms generatedby the plurality of monitoring applications monitoring the networkedcomputing environment.
 13. The system of claim 10, wherein: theaggregated data comprises log file data generated by devices within thenetworked computing environment.
 14. The system of claim 10, wherein:the aggregated data comprises help desk ticket information associatedwith help desk tickets covering performance issues affecting thenetworked computing environment.
 15. The system of claim 10, wherein:the processor configured to output the consolidated alarm bytransmitting a message providing information only associated with thefirst chain of failures to a target recipient.
 16. A computer programproduct, comprising: a non-transitory computer readable storage mediumhaving computer readable program code embodied therewith, the computerreadable program code comprising: computer readable program codeconfigured to detect an alert corresponding with a performance issuewithin a networked computing environment; computer readable program codeconfigured to aggregate data from a plurality of monitoring applicationsmonitoring the networked computing environment, the aggregated datacomprises a plurality of alarms; computer readable program codeconfigured to generate a failure graph based on the aggregated data, thefailure graph comprises a plurality of nodes and a set of directededges, each directed edge of the set of directed edges corresponds witha causal relationship between a pair of the plurality of nodes, thealert corresponds with a root node of the failure graph; computerreadable program code configured to detect a second alert correspondingwith a second performance issue within the networked computingenvironment, the alert corresponds with a first application-levelfailure of a first application within the networked computingenvironment and the second alert corresponds with a secondapplication-level failure of a second application within the networkedcomputing environment different from the first application; computerreadable program code configured to generate a second failure graphbased on the aggregated data, the second alert corresponds with a rootnode of the second failure graph; computer readable program codeconfigured to identify a common leaf node that is common to both thefailure graph and the second failure graph; computer readable programcode configured to identify a first leaf node of the plurality of nodes,the first leaf node corresponds with a root cause of the performanceissue; computer readable program code configured to determine a firstchain of failures corresponding with the first leaf node and the rootnode of the failure graph; computer readable program code configured tosuppress each alarm of the plurality of alarms that is not associatedwith a node in the first chain of failures; and computer readableprogram code configured to output a consolidated alarm correspondingwith the first chain of failures.
 17. The computer program product ofclaim 16, wherein: the consolidated alarm corresponding with the firstchain of failures causes a message specifying the root cause of theperformance issue to be transmitted to a target recipient.